Intrusion detection in computer systems typically starts with designing a computer network to be suitable for collecting data from network traffic. Software filters of various kinds are currently used as tools for detecting an intrusion in different types of computer systems, both network based and host based. To detect an intrusion, network traffic is monitored by software filters and analyzed to find predetermined traffic patterns of interest specific to existing exploits, the patterns customarily referred to as signatures. When the traffic matches a signature, software filters generate alerts. Alerts are often designed to attract attention of security analysts, since signatures have tendency to raise too many false alarms. The analysts then decide whether the alerts indicate an event serious enough to warrant a response. A response may be to shut down a part of the network, to contact the Internet service provider associated with the suspicious traffic, or just to make a note of unusual traffic for future reference.
Intrusion attacks on a computer system are based on a number of software vulnerabilities, of which a buffer overflow attack is the most common.
In computer security, a buffer overflow is an anomalous condition where a large amount of data is attempted to be written into a smaller container beyond the boundaries of a fixed-length buffer. Buffer overflows can be triggered by inputs specifically designed to execute malicious code, also referred to as exploit, or to make a program operate in an unintended way. When a buffer overflow condition is found, hackers normally use it to execute another piece of code, called shellcode, which is a small piece of code used as the payload in exploiting software vulnerability. It typically starts a command shell, from which the attacker can control a compromised computer, making the code to return a shell back to the hacker. A successful buffer overflow attack may give a hacker, or a person who runs the program, same privileges as the vulnerable program, which usually includes administrative privileges.
Development of effective intrusion prevention/detection filters is not an easy task. There are many examples of exploits available for downloading from the Internet, so that even people who do not have professional computer skills are able to run them towards vulnerable hosts. Additionally, those exploits are often obfuscated to bypass network security software. As a result, it is often very hard to detect such exploits. Even after raising an alert, it often takes hours of labor intense analysis to investigate the validity of the alert.
A great variety of exploits can be generated automatically using freely available software tools, such as Metasploit and Canvas, which substantially broadens the scope of the attack. Detection of such exploits is very challenging, since it is not feasible to collect all signatures generated by the exploits.
Many exploits are designed using polymorphic techniques, i.e. each newly generated exploit is encoded differently to avoid being detected by intrusion prevention/detection filters.
In spite of a certain success in developing intrusion prevention/detection filters that are efficient against specific exploits, a need still exists in the computer industry for developing improved methods and systems, which would be exploit nonspecific, and effective in protecting against various types of exploits, as well as developing smart filters therefor.